Short-term Associative Memory

One of the simplest associative memories is the Willshaw Network (Willshaw, Buneman & Longuet-Higgins, 1969). Like other associative networks however (e.g., Hopfield, 1982), it fails completely as a memory device as soon as its capacity is exceeded. Three methods of synaptic change are analysed, decay, ageing and depression, under which this catastrophic failure can be preempted and stability under continuous learning ensured. These methods allow a Willshaw Network to function as a short-term memory, with effective storage of a well-defined number of recent associations, accompanied by the progressive forgetting of older ones. Expressions for the shortterm capacity under each method are obtained in the sparse coding limit and validated via simulation. Learning, retrieval and storage in the Willshaw Network Consider a square Willshaw Network with two, fully-interconnected layers of N cells. Each cell j can be in one of two activity states, aj, firing (aj=1) or quiescent (aj=0). Similarly, a synapse connecting cell j to cell i has two states, Sij, potentiated (Sij=1) or unpotentiated (Sij=0). Synapses are potentiated upon conjoint preand postsynaptic firing. Thus the network learns an association between a pattern of presynaptic activity and a pattern of postsynaptic activity in a simple Hebbian manner. The network can retrieve a postsynaptic activity pattern resembling that previously associated with a given presynaptic activity pattern by feedforward of activity. Specifically, the activity of postsynaptic cell i is determined by thresholding the dendritic sum of impinging presynaptic activity: where Ti is the threshold of cell i. An error arises when the resulting activity of postsynaptic cell i differs from that it possessed in the postsynaptic activity pattern of the previously learned association. If the error rate does not exceed some criterion, that association is deemed stored. Capacity of the standard Willshaw Network Consider the learning of a series of associations between random activity patterns presented over preand postsynaptic cells at discrete times t. Assume each pattern involves M of the N cells firing. If N is large and the firing ratio, F=M/N, is small (the sparse coding limit), then the probability that a synapse is potentiated at time t, or the loading of the network, p(t), is: assuming the network is initially a tabula rasa (i.e., p(0)=0). If the storage criterion is an average of one spurious postsynaptic firing, then, setting all thresholds T=M and making the “unit usage assumption” (Buckingham & Willshaw, 1992), p is effectively constrained by . Willshaw et al. (1969) showed that maximum information efficiency of the network (of ln2) then occurs when p=0.5 and M=ln2(N). Alternatively, using a criterion of fewer than spurious postsynaptic firings, the number of associations stored, or the capacity of the network, c(t), is predicted by: (where is the number of combinations of k in N events). Taking a storage criterion of no more than one spurious firing (L=2), the capacity can be approximated in the sparse coding limit by: *M.R.C. Applied Psychology Unit, 15, Chaucer Road, Cambridge, CB2 2EF, UK email: [email protected] +Centre for Cognitive Science, 2, Buccleuch Place, Edinburgh, EH9 2LD, UK email: [email protected] ai f Sijaj j 1 = N ∑     = f x ( ) 1 0 if if x Ti ≥ x Ti < = p t ( ) 1 1 F − ( ) t − 1 e − xp Ft ( ) ≈ =